feat: Speculative Availability #10563

timwu20 · 2025-12-06T05:20:36Z

Description

Introduces Speculative Availability chunk requests within the Availability Distribution subsystem. In Availability Distibution subsystem we listen on ActiveLeavesUpdate and call Prospective Parachains to gather backable candidates that we can start fetching their erasure encoded chunks from the backing group before the candidates are actually backed on chain. This feature is currently enabled by running the node with the --speculative-availability CLI flag.

Review Notes

Note: This PR is a reimplementation of PR#9444 with less changes to the handling of fetch tasks, overall less diffs, and added CLI flag to enable feature.

Moves request_backale_candidates out of Provisioner subsystem to subystem-util crate to be used as well in Availability Distribution subsystem.
Introduces CoreInfo and CoreInfoOrigin private types in availability-distribution::requester module. CoreInfo is used to create fetch tasks now instead of using available cores directly. We construct CoreInfo instances from calling Prospective Parachains for backable candidates and we denote the origin of these CoreInfo instances with CoreInfo::Scheduled. CoreInfo instances created from candidates that have been backed on chain have CoreInfo::Occupied origin.
Modifies Availability Store subsystem to accept new AvailabilityStoreMessage::NoteBackableCandidates message type.
- This new message type is handled by the Availability Store subsystem to write the meta data to pass validation when accepting the actual chunk for the associated candidate hash. It was discovered during development, that storing the chunk from scheduled/early fetch requests were not being persisted due to not having the meta data previously stored for said candidate.
Includes Zombienet test that utilizes two parachains which support elastic scaling. Assertions are done on the polkadot_parachain_fetched_chunks_total[origin="scheduled"] metric.
--speculative-availability CLI flag support.

TODO

Benchmark with speculative availability enabled.

Checklist

My PR includes a detailed description as outlined in the "Description" and its two subsections above.
My PR follows the labeling requirements of this project (at minimum one label for T required)
- External contributors: Use /cmd label <label-name> to add labels
- Maintainers can also add labels manually
I have made corresponding changes to the documentation (if applicable)
I have added tests that prove my fix is effective or that my feature works (if applicable)

… test

…s, add num_validators attribute

…stribution

cla-bot-2021 · 2025-12-06T05:20:48Z

User @timwu20, please sign the CLA here.

timwu20 · 2025-12-06T05:29:35Z

/cmd prdoc

github-actions · 2025-12-06T05:32:15Z

Command "prdoc" has failed ❌! See logs here

timwu20 · 2025-12-06T05:37:35Z

/cmd label T0-node T8-polkadot

github-actions · 2025-12-06T05:39:51Z

Command "label T0-node T8-polkadot" has failed ❌! See logs here

haikoschol

lgtm, just some comment/logging nits

haikoschol · 2025-12-10T08:06:19Z

polkadot/node/network/availability-distribution/src/requester/mod.rs

+			"Got {} backable candidates", backable.len()
+		);
+
+		// now get the backed candidates corresponding to these candidate receipts


Suggested change

// now get the backed candidates corresponding to these candidate receipts

// now get the backed candidates corresponding to these candidate hashes

since the candidate receipt is part of the data we're retrieving

haikoschol · 2025-12-10T08:22:23Z

polkadot/node/network/availability-distribution/src/requester/mod.rs

+						gum::warn!(
+							target: LOG_TARGET,
+							error = ?err,
+							"Failed to spawn a fetch task"


Suggested change

"Failed to spawn a fetch task"

session_index = ?leaf_session_index,

"Failed to retrieve session info",

haikoschol · 2025-12-10T08:36:38Z

polkadot/node/network/availability-distribution/src/requester/mod.rs

+				if let Some(session_info) = session_info {
+					let num_validators =
+						session_info.validator_groups.iter().fold(0usize, |mut acc, group| {
+							acc = acc.saturating_add(group.len());
+							acc
+						});
+
+					let (tx, rx) = oneshot::channel();
+					sender
+						.send_message(AvailabilityStoreMessage::NoteBackableCandidates {
+							candidate_hashes,
+							num_validators,
+							tx,
+						})
+						.await;
+					rx.await?.map_err(|_| Error::AvailabilityStore)?;
+					self.add_cores(ctx, runtime, leaf, leaf_session_index, scheduled_cores).await?;
+				}


Suggested change

if let Some(session_info) = session_info {

let num_validators =

session_info.validator_groups.iter().fold(0usize, |mut acc, group| {

acc = acc.saturating_add(group.len());

acc

});

let (tx, rx) = oneshot::channel();

sender

.send_message(AvailabilityStoreMessage::NoteBackableCandidates {

candidate_hashes,

num_validators,

tx,

})

.await;

rx.await?.map_err(|_| Error::AvailabilityStore)?;

self.add_cores(ctx, runtime, leaf, leaf_session_index, scheduled_cores).await?;

}

let Some(session_info) = session_info else { continue };

let num_validators =

session_info.validator_groups.iter().fold(0usize, |mut acc, group| {

acc = acc.saturating_add(group.len());

acc

});

let (tx, rx) = oneshot::channel();

sender

.send_message(AvailabilityStoreMessage::NoteBackableCandidates {

candidate_hashes,

num_validators,

tx,

})

.await;

rx.await?.map_err(|_| Error::AvailabilityStore)?;

self.add_cores(ctx, runtime, leaf, leaf_session_index, scheduled_cores).await?;

haikoschol · 2025-12-10T08:50:14Z

polkadot/node/subsystem-types/src/messages.rs

 		tx: oneshot::Sender<Result<(), ()>>,
 	},

+	/// Note that a candidate is backable. Only used by Speculative Availability to signal to


Consider updating the comments to use the plural since the message includes multiple candidate hashes.

haikoschol · 2025-12-10T08:57:33Z

polkadot/zombienet-sdk-tests/tests/functional/speculative_availability_requests.rs

+	// given when speculative availability is requested on active leaves update, there may not be
+	// any backable candidates from Prospective Parachains at that the time.


Suggested change

// given when speculative availability is requested on active leaves update, there may not be

// any backable candidates from Prospective Parachains at that the time.

// Given that speculative availability is requested on active leaves update, there may not be

// any backable candidates from Prospective Parachains at that time.

timwu20 added 13 commits December 5, 2025 14:20

core info, get backable candidates, hack av-store

dcd9d9a

add new av store message type, send message for early fetches

9fedb16

remove duplicate function

e2ee38b

remove hack when querying chunk

bf6b456

clean up error handling, revise fetching backable candidates

bff8238

add CoreInfoOrigin to request types

3b8c5e2

add wip zombienet test for speculative availability

ac4f97b

add origin metric label to fetched_chunks, assert on said label in ZN…

db5b912

… test

revise NoteBackableCandidate msg to handle HashSet of candidate hashe…

cd93f03

…s, add num_validators attribute

add speculative-availability cli flag, integrate into availability di…

6e087a7

…stribution

code cleanup

9f930cd

fix zn test

e73deba

fix benchmark

612d5d2

timwu20 marked this pull request as draft December 6, 2025 05:20

cargo fmt

91c62dd

fix polkadot_service::NewFullParams

59ce4bb

kmk142789 approved these changes Dec 6, 2025

View reviewed changes

fix provisioner and availability-distribution tests

da460f7

timwu20 marked this pull request as ready for review December 7, 2025 20:32

haikoschol approved these changes Dec 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Speculative Availability #10563

feat: Speculative Availability #10563

timwu20 commented Dec 6, 2025 •

edited

Loading

Uh oh!

cla-bot-2021 bot commented Dec 6, 2025

Uh oh!

timwu20 commented Dec 6, 2025

Uh oh!

github-actions bot commented Dec 6, 2025

Uh oh!

timwu20 commented Dec 6, 2025

Uh oh!

github-actions bot commented Dec 6, 2025

Uh oh!

haikoschol left a comment

Uh oh!

haikoschol Dec 10, 2025

Uh oh!

haikoschol Dec 10, 2025

Uh oh!

haikoschol Dec 10, 2025

Uh oh!

haikoschol Dec 10, 2025

Uh oh!

haikoschol Dec 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	// now get the backed candidates corresponding to these candidate receipts
	// now get the backed candidates corresponding to these candidate hashes

	"Failed to spawn a fetch task"
	session_index = ?leaf_session_index,
	"Failed to retrieve session info",

		// given when speculative availability is requested on active leaves update, there may not be
		// any backable candidates from Prospective Parachains at that the time.

feat: Speculative Availability #10563

Are you sure you want to change the base?

feat: Speculative Availability #10563

Conversation

timwu20 commented Dec 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Review Notes

TODO

Checklist

Uh oh!

cla-bot-2021 bot commented Dec 6, 2025

Uh oh!

timwu20 commented Dec 6, 2025

Uh oh!

github-actions bot commented Dec 6, 2025

Uh oh!

timwu20 commented Dec 6, 2025

Uh oh!

github-actions bot commented Dec 6, 2025

Uh oh!

haikoschol left a comment

Choose a reason for hiding this comment

Uh oh!

haikoschol Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

haikoschol Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

haikoschol Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

haikoschol Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

haikoschol Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

timwu20 commented Dec 6, 2025 •

edited

Loading